Clean up CeleryExecutor to use workload terminology and typing#63888
Open
SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Open
Clean up CeleryExecutor to use workload terminology and typing#63888SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Conversation
…ing to align with the workload-based executor model.
8e2dea8 to
29a855f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR is a follow-up for PR #61153. It performs a non-functional cleanup of the Celery executor to better align it with the workload-based execution model being introduced in Airflow 3.2.
The changes include:
WorkloadKey,WorkloadInCelery,workloads.All).Method renames were applied selectively:
BaseExecutor/ scheduler-invoked interfaces were left unchanged to preserve compatibilityNo behavioral or functional changes are introduced.
Rationale
The Celery executor already supports workload-based execution, but parts of the implementation still reflect legacy task-centric terminology. This PR standardizes naming, typing, and documentation to improve consistency and readability.
Method names were updated where safe (i.e. internal helpers not part of public or inherited interfaces). Methods that mirror
BaseExecutor, are invoked by the scheduler, or are tightly coupled to Celery internals were intentionally left unchanged to avoid breaking contracts.Variables were renamed from “task” to “workload” where they represent Airflow execution units. Fields inherited from
BaseExecutor(e.g.queued_tasks) and Celery-specific concepts (e.g.task_id) were preserved to maintain compatibility and clarity.Docstrings and comments were updated to consistently describe workloads while still distinguishing between Airflow workloads and Celery tasks where necessary.
Typing was improved by preferring workload-based types while retaining legacy aliases required for compatibility across Airflow versions.
Notes
The
CeleryKubernetesExecutorhas not been altered (save for atype: ignoreto satisfy mypy) in this PR as it is now deprecated according to Issue Executor Callback support #62887Metrics (
Stats.incr) and configuration paths were intentionally left unchanged, as they form part of Airflow’s external/operational interface (e.g. dashboards, alerts, and user configs). Renaming these could introduce unintended breaking changes or disrupt existing observability. This change is therefore scoped to internal naming, typing, and documentation updates only, without altering behavior or externally visible contracts.Related